Multimodal signal processing is more than simply "putting together" text, audio, images and video; it is the integration and interaction among these different media that creates new systems and new research challenges and opportunities. Unimodal analysis of signals can deliver acceptable performance levels only in benign situations; the performance decreases rapidly when countermeasures are taken. For example, person authentication systems useful in security, access control and surveillance applications do not perform well when subjects age, when the video resolution is inadequate, or poor lighting conditions are present. Many of these difficulties can be overcome by adding an audio signature along with the video.
In multimodal communications where humans speech is involved, audio-visual interaction is particularly significant. Human perception of speech is bimodal in that acoustic speech can be affected by visual cues from lip movement. Due to the bimodality in speech perception, audio-visual interaction is an important design factor for multimodal communication systems, such as video telephony and video conferencing. A prime example of this interaction is lip or speech reading. It is used by the hearing-impaired for enhancing their speech understanding capability but also by every normal hearing person to some extent, in particular in noisy environments.
One key issue in bimodal speech analysis and synthesis is the establishment of the mapping between acoustic and visual parameters. A novel approach for establishing this mapping was developed during our previous funding period. Our current work addresses two inter-related problems. First, the synthesis of articulatory parameters for an MPEG-4 facial animation model is being considered. Second, we are concerned with the task of robust speech recognition. Fusing these two areas will impact the fields of very low bit-rate coding of speech and images, speech and text driven facial animation parameters, speech and text driven facial animation of synthetic actors (i.e. avators) and audio-visual speech recognition.